Electronic device and method for controlling electronic device

ABSTRACT

An electronic device, for supporting image capturing by an imaging apparatus, includes at least one processor and/or at least one circuit which function as: a generating unit configured to generate a first object in a virtual space corresponding to a current image capturing range of the imaging apparatus, and a second object in a virtual space corresponding to an image capturing range in a case where the imaging apparatus has a predetermined position and a predetermined orientation; and a display control unit configured to display the first object and the second object.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an electronic device and a method forcontrolling the electronic device.

Description of the Related Art

In a camera or a camera installed in a smartphone, functions to supportimage capturing have been included. The functions to support imagecapturing include, for example, a function to display grid or horizontallines on a live view (LV) screen, a peeking function to show a user alocation where a subject displayed on the LV screen is focused, and afunction to indicate a destination to which the camera moves inpanoramic image capturing.

Also in recent years, techniques such as augmented reality (AR) andmixed reality (MR) are used in a head mount display (HMD), a cameraequipped smartphone, a tablet terminal, and the like.

For example, as a technique to support determining an angle of view,Japanese Patent Application Publication No. 2020-167499 discloses anapparatus that supports fixed-point image capturing. Specifically, theapparatus according to Japanese Patent Application Publication No.2020-167499 generates AR objects of a position of a camera and an imagecapturing range at the time when a photograph was taken in the past, andsuperimposes and displays the generated AR objects on an LV screen ofthe camera of the real object. By moving the camera of the real objectand overlaying the same on the camera of the AR object displayed on theLV screen, and also by matching the image capturing range with the imagecapturing range of the AR object, the user can capture an image at thesame angle of view of a photograph taken in the past.

Japanese Patent Application Publication No. 2011-193286 discloses anapparatus that supports panoramic image capturing. Specifically, theapparatus according to Japanese Patent Application Publication No.2011-193286 displays, on the LV screen, a two-dimensional object, toguide an optimum moving speed of the apparatus for panoramic imagecapturing.

However, even if the camera of an AR object is displayed on the LVscreen of the camera of the real object, in some cases it may bedifficult to overlay the camera of the real object accurately on thecamera of the AR object. Furthermore, if the shape of the camera of theAR object is different from the shape of the camera used for imagecapturing, it is difficult to accurately match the position andorientation of the camera of the real object with the camera of the ARobject.

Furthermore, in the case where a two-dimensional object for guiding themoving speed is displayed on the LV screen, movement of a camera on thetwo-dimensional plane that is parallel with the imaging plane can besupported, but supporting the image capturing in a three-dimensionalspace is difficult.

SUMMARY OF THE INVENTION

The present invention provides an electronic device that makesdetermination of an appropriate angle of view easier, and supports imagecapturing in the three-dimensional space.

The electronic device of the present invention is an electronicapparatus for supporting image capturing in an imaging apparatus, theelectronic device including at least one processor and/or at least onecircuit which function as: a generating unit configured to generate afirst object in a virtual space corresponding to a current imagecapturing range of the imaging apparatus, and a second object in avirtual space corresponding to an image capturing range in a case wherethe imaging apparatus has a predetermined position and a predeterminedorientation; and a display control unit configured to display the firstobject and the second object.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are diagrams exemplifying an external view of an imagingapparatus;

FIG. 2 is a diagram exemplifying an external view of an HMD;

FIG. 3 is a diagram depicting a hardware configuration example of theimaging apparatus of Embodiment 1;

FIG. 4 is a flow chart exemplifying image capturing support processingof Embodiment 1;

FIGS. 5A to 5C are diagrams for describing path information;

FIGS. 6A and 6B are diagrams for describing path information;

FIG. 7 is an example of generating an imaging object and a supportingobject according to Embodiment 1;

FIGS. 8A to 8C are display example of an imaging object and a supportingobject on an LV screen;

FIG. 9 is a flow chart exemplifying image capturing support processingof Embodiment 2;

FIG. 10 is an example of generating an imaging object and a supportingobject according to Embodiment 2;

FIG. 11 is a diagram depicting hardware configuration examples of animaging apparatus and an HMD of Embodiment 3; and

FIGS. 12A and 12B are flow charts exemplifying image capturing supportprocessing of Embodiment 3.

DESCRIPTION OF THE EMBODIMENTS

Configurations of an imaging apparatus and an HMD will be described asembodiments of an electronic device according to the present invention,with reference to FIGS. 1A, 1B and FIG. 2 . The configurations describedin FIGS. 1A, 1B and FIG. 2 are the same for all the embodimentsdescribed herein below.

FIGS. 1A and 1B are diagrams exemplifying an external view of an imagingapparatus. The imaging apparatus 1 may be integrated into the electronicdevice, or may be separate from the electronic device. FIGS. 1A and 1Bindicate a front face and a rear face of the imaging apparatus 1respectively. The imaging apparatus 1 is an apparatus having an imagepickup element, such as a camera, a smartphone and a tablet terminal.The imaging apparatus 1 includes an imaging unit 11, a display unit 12and an operation unit 13.

At least one imaging unit 11 is mounted on at least one of the frontface and the rear face of the imaging apparatus 1. The display unit 12is a display having a touch panel, and displays various images. Thetouch panel of the display unit 12 accepts operations performed by usinga finger of the user, a stylus pen, or the like. The operation unit 13is a member to input operations that the user performs via a button,dial, and the like. One or more operation units 13 may be disposed onthe imaging apparatus 1, or the operation unit 13 may be implemented bya touch panel of the display unit 12. The display unit 12 is an exampleof the “display control unit”.

FIG. 2 is a diagram exemplifying an external view of an HMD2. The HMD2exemplified in FIG. 2 is an optical see-through type HMD. The HMD2 is ahead mounted type display device, and is also called AR glasses. TheHMD2 includes a frame and two display units (display unit 21 a anddisplay unit 21 b). The frame includes a rim 25 and temples 26 a and 26b which are bonded to each side of the rim 25. The display units 21 aand 21 b are bonded to one side of the rim 25.

The light from a display element (not illustrated) of the display unit21 a is guided to the right eye of a user wearing the HMD2 via animage-projecting unit 22 a and a light-guiding unit 23 a. In the samemanner, the light from a display element (not illustrated) of thedisplay unit 21 b is guided to the left eye of the user wearing the HMD2via an image-projecting unit 22 b and a light-guiding unit 23 b.

The user can recognize the images displayed on the display unit 21 a andthe display unit 21 b and the incident light coming from the front sideof the HMD2 simultaneously. The HMD2 may have two imaging units (imagingunit 24 a and imaging unit 24 b). The imaging unit 24 a and the imagingunit 24 b capture images of the surroundings, including the front areaof the HMD. In this description, the HMD2 is assumed to be an opticalsee-through type HMD using a transparent display, but may be a videosee-through type HMD using an opaque display.

The imaging apparatus 1 or the HMD2 according to each embodimentdescribed below recognizes the surrounding space using an equippedimaging unit or the like, whereby various virtual objects matching withthe surrounding space can be presented to the user as three-dimensionalinformation. By presenting the information using a virtual space orvirtual objects, information in the depth direction, which has beendifficult for the user to recognize, can be provided to the user. Thevirtual objects mapped in a real space can be used for various imagingcapturing support units.

Embodiment 1 and Embodiment 2 are embodiments to support image capturingby the imaging apparatus 1 or the HMD2 when an image is captured usingone of the imaging apparatus 1 and the HMD2. Embodiment 3, on the otherhand, is an embodiment to support image capturing by the imagingapparatus 1 in the case where the imaging apparatus 1 and the HMD2 areoperated in tandem.

Embodiment 1

Embodiment 1 is an embodiment to support the movement of the imagingapparatus 1 or the HMD2 by displaying a moving destination, in order tocapture an image at an optimum angle of view using either one of theimaging apparatus 1 and the HMD2. In the following description, a caseof supporting image capturing performed by the imaging apparatus 1 willbe described.

FIG. 3 is a diagram depicting a hardware configuration example of theimaging apparatus 1 according to Embodiment 1. The imaging apparatus 1includes an arithmetic unit 101, an image processing unit 102, a primarystorage unit 104, a secondary storage unit 105, a display unit 106, animaging unit 107, an operation unit 108 and a three-dimensional spacerecognizing unit 109. Each composing element of the imaging apparatus 1is interconnected via a bus.

the arithmetic unit 101 is a central processing unit (CPU), for example,and controls the imaging apparatus 1 in general. The arithmetic unit 101may include a hardware circuit that implements functions executed by theimaging apparatus 1. The image processing unit 102 generates an object(imaging object) that indicates an imaging angle of view in a virtualspace recognized by the three-dimensional space recognizing unit 109,and a supporting object that supports image capturing with the imagingapparatus 1 by the user.

The imaging object is an AR object in a virtual space, which correspondsto a current image capturing range of the imaging apparatus. Thesupporting object is an AR object in a virtual space, which correspondsto an image capturing range in a case where the imaging apparatus 1 hasa predetermined position and a predetermined orientation. The supportingobject is an AR object to present an ideal moving destination of animaging object, in order to move the imaging apparatus 1 with idealcamera work. The imaging object corresponds to the “first object”. Thesupporting object corresponds to the “second object”.

The image processing unit 102 generates a supporting object such thatthe movement of the imaging apparatus 1 is controlled by the useroverlaying the imaging object with a supporting object. By moving theimaging apparatus 1 so that the imaging object is overlaid on thesupporting object, the user can capture an image at a constant speed, orcapture an image according to a regular path. The image processing unit102 is an example of the “generating unit”.

The primary storage unit 104 is a dynamic random access memory (DRAM),for example, and temporarily stores image data. The secondary storageunit 105 is a flash memory, for example, and records captured images andthe like.

The display unit 106 is a display to display captured images and othervisual objects. The display unit 106 displays an imaging object and asupporting object superimposed on an image capturing a real space. Theimaging unit 107 converts light into electric signals. The operationunit 108 accepts operation from the user. The display unit 106, theimaging unit 107 and the operation unit 108 correspond to the displayunit 12, the imaging unit 11 and the operation unit 13 indicated in FIG.1B respectively.

The three-dimensional space recognizing unit 109 detects a position ofan object in and a distance to the object in a real space, and acquiresinformation on a virtual space. The three-dimensional space recognizingunit 109 can recognize the real three-dimensional space using aplurality of cameras, for example. The three-dimensional spacerecognizing unit 109 is an example of the “acquiring unit”. Theprocessing performed by the three-dimensional space recognizing unit 109may be executed by the imaging unit 107.

FIG. 4 is a flow chart exemplifying image capturing support processing400 according to Embodiment 1. The image capturing support processing400 is processing to support image capturing by the user, by generatingand displaying an imaging object and a supporting object on the LVscreen. The imaging object and the supporting object are displayedsuperimposed on the image capturing the real space. In Embodiment 1, thesupporting object is generated based on path information which indicatesthe future movement of the imaging apparatus 1, such as the moving speedand path thereof.

The image capturing support processing 400 is started when the imagingapparatus 1 enters an image capturing mode by the user operating theoperation unit 108 of the imaging apparatus 1, for example. Thearithmetic unit 101 of the imaging apparatus 1 executes programs foreach composing element to perform each processing step of the imagecapturing support processing 400.

In S401, the image processing unit 102 acquires the path informationwhich is used for generating the supporting object. The path informationis information to indicate future movement of the image capturing range(camera work) of the captured image. The path information includesinformation to determine the position, orientation (direction componentfrom the position of the supporting object), moving speed and path ofthe supporting object in the virtual space.

The supporting object is an AR object in a virtual space, whichcorresponds to an image capturing range in a case where the imagingapparatus is at a predetermined position and is in a predeterminedorientation. Based on the acquired path information, the imageprocessing unit 102 can determine the predetermined position andpredetermined orientation. The image processing unit 102 may acquire thepath information stored in the secondary storage unit 105 in advance, ormay acquire the path information inputted by the user via the operationunit 108.

The path information will now be described with reference to FIGS. 5A to5C, FIG. 6A and FIG. 6B. The path information is information to indicatemovement of the imaging apparatus 1, and is information indicating, forexample, movement when the imaging apparatus 1 moves straight ahead at aconstant speed, or movement when the imaging apparatus 1 rotates in thehorizontal direction around the imaging apparatus 1 or the user. Thepath information may be information indicating movement when the imagingapparatus 1 rotates at a constant speed around the optical axis of lightentering the imaging unit 107 (imaging axis of imaging apparatus 1), ormovement when the imaging apparatus 1 rotates around a subject.

FIG. 5A is an example when a person (user capturing the image) is movingstraight ahead at a constant speed. When the user overlays an imagingobject 510 on a supporting object 511, the display unit 106 displays asupporting object 512 ahead of the supporting object 511. Then when theuser overlays the imaging object 510 on the supporting object 512, thedisplay unit 106 displays a supporting object 513 ahead of thesupporting object 512.

FIG. 5B is an example when a person moves in the lateral direction at aconstant speed. When the user overlays an imaging object 520 on thesupporting object 521, the display unit 106 displays a supporting object522 in the lateral direction of the supporting object 521.

FIG. 5C is an example when the imaging apparatus 1 is rotated in thehorizontal direction at a constant speed around the imaging apparatus 1.When the user overlays an imaging object 530 on a supporting object 531,the display unit 106 displays a supporting object 532 in a directionwhere the imaging apparatus 1 rotates. Then when the user overlays theimaging object 530 on the supporting object 532, the display unit 106displays a supporting object 533 in a direction where the imagingapparatus 1 rotates.

FIG. 6A is an example when the imaging apparatus 1 is rotated at aconstant speed around the image capturing axis of the imaging apparatus1. When the user overlays an imaging object 610 on a supporting object611, the display unit 106 displays a supporting object 612 in adirection where the imaging apparatus 1 rotates around the imagecapturing axis. Then when the user overlays the imaging object 610 onthe supporting object 612, the display unit 106 displays a supportingobject 613 in a direction where the imaging apparatus 1 is in adirection where the imaging apparatus 1 rotates around the imagecapturing axis.

FIG. 6B is an example when the imaging apparatus 1 is rotated at aconstant speed around a subject. When the user overlays an imagingobject 620 on a supporting object 621, the display unit 106 displays asupporting object 622 in a direction where the imaging apparatus 1rotates around the subject.

The supporting objects may be displayed one at a time in accordance withthe movement of the imaging apparatus 1 based on the path information.Further, a plurality of supporting objects may be displayed such thatthe predetermined position and the predetermined orientation are aposition and an orientation of the imaging apparatus 1 at each of theplurality of future timings based on the path information. Thesupporting object displayed in the past may not be displayed in the casewhere the difference in position or orientation from that of the imagingobject is a threshold or more, or may be continuously displayed.

The path information is not limited to the speed or path of the imagingapparatus 1, but may be information to indicate a movement to positionthe subject at a predetermined position (e.g. center or edge of imagecapturing range), or may be a movement to place the front face of thesubject within the image capturing range.

In S402 in FIG. 4 , the three-dimensional space recognizing unit 109acquires the position of the object in the real space and the distancefrom the imaging apparatus 1 to the object as virtual space information,and stores the information in the primary storage unit 104.

In S403, the image processing unit 102 generates an imaging object,which indicates an image capturing range where the imaging unit 107 iscaptured an image, in the virtual space recognized by thethree-dimensional space recognizing unit 109. The image processing unit102 adds information on the generated imaging object to the virtualspace information that is held in the primary storage unit 104. Theimaging object is an AR object which indicates a position and theorientation corresponding to the image capturing range of the imagingapparatus 1.

In the virtual space information that is held in the primary storageunit 104, the image processing unit 102 searches a portion matching withthe image being captured by the imaging apparatus 1, using templatematching, for example. The image processing unit 102 can generate theimaging object based on the position and orientation of the searchedportion matching with the image capturing in the virtual space.

The shape of the imaging object may be a shape that can uniquelydetermine a position and orientation, such as a frame, rectangularparallelepiped, cursor, or points indicating four corners of the imagecapturing range. The position of the imaging object in the depthdirection may be a position at a focal distance which is the bestfocused, for example.

The position of the imaging object in the depth direction may be aposition that is distant from the imaging apparatus 1 by a predetermineddistance, and the size of the imaging object may be set to be variable.For example, in the case where the shape of the imaging object is aframe, the distance from the imaging apparatus 1 to the frame may befixed to 1 m, which is predetermined, and the size of the frame may beset to be variable. The distance from the imaging apparatus 1 to theframe may be a distance from the imaging apparatus 1 to the center ofthe frame, for example.

The position of the imaging object in the depth direction may be setsuch that the size of the imaging object has a predetermined size, andthe distance from the imaging apparatus 1 to the imaging object is setto be variable. For example, in a case where the shape of the imagingobject is a frame, the size of the frame may be fixed to a predeterminedsize, such as 30 cm of the diagonal line of the frame in the real space,and the distance from the imaging apparatus 1 to the frame may be set tobe variable.

In the case of a small space where an obstacle exists in front of theimaging apparatus 1, for example, the image processing unit 102 maygenerate the imaging object such that the imaging object is displayed ata position closer to the imaging apparatus 1 than the obstacle. Theimage processing unit 102 can change the display position of the imagingobject in accordance with the surrounding space.

In the case of the above mentioned example, the position and orientationof the imaging object are acquired based on the image capturing rangewhere the imaging unit 107 is capturing an image. The position andorientation of the imaging object may also be acquired from the positionand orientation of the imaging apparatus 1 in the real space.

In S404, the image processing unit 102 generates a supporting object,which is an AR object to support image capturing. The image processingunit 102 generates the supporting object based on the path informationacquired in S401.

The path information is information to indicate the movement of theimage capturing range of the capturing image (camera work). For example,in a case where the path information is information to indicate themovement when the imaging apparatus 1 moves straight ahead at a constantspeed, the image processing unit 102 generates the supporting object sothat the current imaging object can be displayed at a position where theimaging apparatus 1 is moved straight ahead according to the orientationof this imaging object. Specifically, the image processing unit 102 maygenerate the supporting object such that the orientation of the imagingapparatus 1 is the predetermined orientation, and the position, wherethe imaging apparatus 1 is moved straight ahead in the predeterminedorientation for one second at a constant speed, is the predeterminedposition.

The image processing unit 102 can generate the supporting object at aposition to which the current imaging object is moved in accordance withthe movement indicated in the path information. The image processingunit 102 may generate the supporting object such that the relativerelationship between the predetermined position of the imaging apparatus1 and the position of the supporting object becomes the same as therelative relationship between the current position of the imagingapparatus 1 and the position of the imaging object. The image processingunit 102 adds the information on the generated supporting object to thevirtual space information that is held in the primary storage unit 104.

FIG. 7 is an example of generating an imaging object and a supportingobject. FIG. 7 is a schematic diagram depicting a state whereinformation on an imaging object 702 and information on a supportingobject 703 are added to the information on a virtual space 700.

In S405 of FIG. 4 , the display unit 106 superimposes the imaging objectand the supporting object that are held in the primary storage unit 104on the image capturing the real space, and displays the generated imageon the LV screen. The superimposing processing may be performed by theimage processing unit 102.

In the superimposing processing, the display unit 106 first acquiresinformation on a portion of the virtual space 700 corresponding to theimage generated by the imaging unit 107 capturing the real space. Thenthe display unit 106 superimposes the imaging object 702 and thesupporting object 703 existing in the virtual space 700 on correspondingpositions in the image capturing the real space.

In S406, the arithmetic unit 101 determines whether the user instructedthe operation unit 108 to end the image capturing mode. The imagecapturing support processing 400 ends if the end of the image capturingmode was instructed (S406: Yes). If the end of the image capturing modewas not instructed (S406: No), processing returns to S402, and theprocessing steps from S402 to S406 are repeated.

The processing steps from S402 to S406 are processing steps which areexecuted for each frame of the moving image that is captured by theimaging unit 107. However, the processing steps from S402 to S406 may beexecuted for a plurality of frames (e.g. five frames) at a time, as longas the supporting object can be displayed in such a way that the imagecapturing by the user is supported.

FIGS. 8A to 8C are display examples of an imaging object and asupporting object on the LV screen. FIGS. 8A to 8C indicate exampleswhen an imaging object and a supporting object are superimposed on animage capturing the real space, and are displayed on an LV screen 800(display unit 106).

FIG. 8A is an example when an imaging object 801 having a frame shapeand a supporting object 802 having a frame shape exist on the LV screen800 at the same depth and in the same orientation. The user canimplement an appropriate image capturing angle of view or camera work bymoving the imaging apparatus 1 such that the imaging object 801, whichmoves interlocking with the imaging apparatus 1, is overlayed on thesupporting object 802.

FIG. 8B is an example when an imaging object 803 having a frame shapeand a supporting object 804 having a frame shape exist on the LV screen800 at different depths and in different orientations. The user canimplement an appropriate image capturing angle of view or camera work bymoving the imaging apparatus 1 such that the imaging object 803, whichmoves interlocking with the imaging apparatus 1, is overlaid on thesupporting object 804.

FIG. 8C is an example when an imaging object 805 having a rectangularparallelepiped shape and a supporting object 806 having a rectangularparallelepiped shape exist on the LV screen 800 at different depths andin different orientations. The user can implement an appropriate imagecapturing angle of view or camera work by moving the imaging apparatus 1such that the imaging object 805, which moves interlocking with theimaging apparatus 1, is overlaid on the supporting object 806.

In the above example, a case of supporting the image capturing by theuser using the imaging apparatus 1 was described, but the presentembodiment may be implemented by an HMD2 having a similar configurationsas the configuration of the imaging apparatus 1 indicated in FIG. 3 .

The supporting object may be constantly displayed while an image iscaptured in the imaging mode, or may be displayed when a predeterminedcondition is satisfied. The predetermined condition is, for example, acase where either the difference in positions or the difference inorientations of the imaging object and the supporting object is athreshold value or more. The predetermined condition may also be a casewhere the moving speed of the imaging apparatus 1 is not constant, or acase where the path of the movement of the imaging apparatus 1 isirregular.

When a predetermined condition is satisfied, the display unit 106 maynotify the user that the imaging apparatus 1 is not moving in accordancewith the path information by temporarily displaying the supportingobject. In the case where a predetermined condition is satisfied and theimaging apparatus 1 is moving in accordance with the path information,the display unit 106 may stop display of the supporting object.

The imaging object and the supporting object are generated by the imageprocessing unit 102 of the imaging apparatus 1, but a cloud-basedexternal device may perform this generation processing. In this case,the imaging apparatus 1 includes a communication unit that cancommunicate via a cloud, and send the information used for generatingthe imaging object and the supporting object. The information to be sentvia cloud is a captured image by the imaging apparatus 1, pathinformation to indicate the movement of the imaging apparatus 1, andvirtual space information, for example.

The imaging apparatus 1 may receive (acquire) the imaging object and thesupporting object generated via a cloud, and display the objects on theLV screen superimposed on an image capturing a real space. By having anexternal device generate the AR content, such as the imaging object andthe supporting object, the imaging apparatus 1 need not include a highperformance large scale SoC, which makes it possible to downsize theimaging apparatus 1.

According to the above mentioned imaging apparatus 1 or the HMD2, theuser can easily determine an appropriate angle of view or implementappropriate camera work, by moving the imaging apparatus 1 or the HMD2such that the imaging object displayed on the LV screen is overlaid onthe supporting object.

Embodiment 2

In Embodiment 1, the image processing unit 102 generates the supportingobject based on the path information and current image capturing range,independently from the movement of the imaging object. In Embodiment 2,on the other hand, the supporting object is generated reflecting themovement of the imaging object in real-time. Therefore the imagingapparatus 1 can generate an optimum supporting object considering themovement of the user.

The hardware configuration of the imaging apparatus 1 according toEmbodiment 2 is the same as the hardware configuration of the imagingapparatus 1 according to Embodiment 1 indicated in FIG. 3 , hencedescription thereof is omitted. Just like Embodiment 1, Embodiment 2 maybe implemented by an HMD2 which has a similar configuration as theconfiguration of the imaging apparatus 1 indicated in FIG. 3 .

FIG. 9 is a flow chart exemplifying an image capturing supportprocessing 900 according to Embodiment 2. The image capturing supportprocessing 900 is a processing to generate a supporting object based onthe movement of an imaging object.

In each processing step of the image capturing support processing 900, asame processing step as the image capturing support processing 400 ofEmbodiment 1 indicated in FIG. 4 is denoted with a same reference sign,and detailed description thereof is omitted. Processing steps S901, S902and S903, which are different from the image capturing supportprocessing 400 of Embodiment 1, will be described.

The image capturing support processing 900 is started, for example, whenthe user operates the operation unit 108 of the imaging apparatus 1, andthe imaging apparatus 1 enters the image capturing mode. The arithmeticunit 101 of the imaging apparatus 1 executes a program for eachcomposing element to perform each processing step of the image capturingsupport processing 400.

In S402, the three-dimensional space recognizing unit 109 acquires thevirtual space information just like Embodiment 1. In S403, the imageprocessing unit 102 generates an imaging object which indicates an imagecapturing range where the imaging unit 107 is capturing an image in thevirtual space.

In S901, it is determined whether this is the first loop of the imagecapturing support processing. In Embodiment 2, the image processing unit102 generates a supporting object based on the movement of a pluralityof imaging objects. In the first loop, only one imaging object isgenerated, hence the image processing unit 102 generates more imagingobjects.

If this is the first loop (S901: Yes), processing returns to S402. Byrepeating the processing steps S402 and S403, the image processing unit102 can generate a plurality of imaging objects having a timedifference. If this is the second or later loop (S901: No), processingadvances to S902.

In S902, the image processing unit 102 generates a supporting object,which is an AR object to support image capturing, from two imagingobjects generated for the current frame and previous frame. Theinformation on the generated supporting object is added to the virtualspace information that is held in the primary storage unit 104.

FIG. 10 is an example of generating an imaging object and a supportingobject. FIG. 10 is a schematic diagram depicting a state whereinformation on an imaging object 1001 of the previous frame, an imagingobject 1002 of the current frame, and a supporting object 1003 is addedto the information on the virtual space 1000. The previous frame is notlimited to the frame immediately before the current frame, but may be aframe that precedes the current frame by a plurality of frames.

Specifically, the supporting object can be generated by the followingmethod. The image processing unit 102 acquires the change amounts of theposition, orientation, moving speed, and the like when the imagingobject 1001 in the previous frame moved to the imaging object 1002 inthe current frame in the virtual space 1000. The change amounts are themoving amount, rotation amount, acceleration and the like from theimaging object 1001 to the imaging object 1002.

Based on the acquired change amounts, the image processing unit 102estimates the position and orientation of the imaging object in the nextframe by the position and orientation of the supporting object 1003. Theimage processing unit 102 adds the information on the estimatedsupporting object 1003 to the information on the virtual space 1000.

Based on the position and orientation of the imaging object 1001 in thepast, and the position and orientation of the current imaging object1002, the image processing unit 102 can determine the predeterminedposition and the predetermined orientation of the imaging apparatus 1,and generate the supporting object.

In S903, the display unit 106 superimposes the imaging object of thecurrent frame and the estimated supporting object on the image capturingthe real space, and displays the objects on the LV screen. Thesuperimposing processing may be performed by the image processing unit102. The processing in S406 is the same as the processing of Embodiment1 indicated in FIG. 4 .

The image processing unit 102 can generate an imaging object in thevirtual space corresponding to the image capturing range in the casewhere the imaging apparatus 1 has the position and orientation in thepast. The display unit 106 may display the imaging object in the past onthe LV screen along with the supporting object and the current imagingobject. The imaging object in the past corresponds to the “thirdobject”.

In the example described above, the imaging apparatus 1 supports theimage capturing by the user, but the present embodiment may beimplemented by an HMD2 having a configuration similar to theconfiguration of the imaging apparatus 1 indicated in FIG. 3 .

While capturing the image in the imaging mode, the supporting object maybe displayed constantly so as to move in real-time, or may be displayedonly when a predetermined condition is satisfied. The predeterminedcondition is, for example, a case where the moving speed of the imagingapparatus 1 is not constant, or a case where the path of the movement ofthe imaging apparatus 1 is irregular. The display unit 106 can notifythe user that the imaging apparatus 1 is not moving at a constant speed,for example, by temporarily displaying the supporting object when thepredetermined condition is satisfied.

The imaging object and the supporting object are generated by the imageprocessing unit 102 of the imaging apparatus 1, but may also begenerated via a cloud. The imaging apparatus 1 may receive (acquire) theimaging object and the supporting object generated via a cloud andsuperimpose and display the objects on the LV screen.

In the example described above, the supporting object is generated basedon the change amount of the imaging object, but may be generated basedon the change amount of the position, orientation, moving speed and thelike of the imaging apparatus 1 when images in the previous frame andcurrent frame are captured. Based on the changes between the previousposition and orientation of the imaging apparatus 1 in the past, and thecurrent position and orientation of the imaging apparatus 1, the imageprocessing unit 102 can determine the predetermined position and thepredetermined orientation of the imaging apparatus 1, and generate thesupporting object.

The above mentioned imaging apparatus 1 or HMD2 can generate asupporting object to determine an appropriate angle of view consideringthe movement of the user. By moving the imaging apparatus 1 or the HMD2so that the imaging object is overlaid on the supporting object, theuser can move the imaging apparatus 1 at a constant speed or rotate theimaging apparatus 1 at a constant angular velocity in accordance withthe movement of the user.

Embodiment 3

Embodiment 1 and Embodiment 2 are embodiments to support image capturingby the user using either the imaging apparatus 1 or the HMD2. Embodiment3, on the other hand, is an embodiment where the imaging apparatus 1 andthe HMD2 are operated in tandem. In Embodiment 3, the imaging apparatus1, which specializes in imaging functions, and the HMD2, which candisplay AR objects in a wide range, can be used utilizing theirrespective advantages.

FIG. 11 is a diagram depicting hardware configuration examples of animaging apparatus 11 and an HMD2 according to Embodiment 3. The imagingapparatus 11 according to Embodiment 3 includes an arithmetic unit 101,an image processing unit 102, a primary storage unit 104, a secondarystorage unit 105, an imaging unit 107, and an operation unit 108, justlike the imaging apparatus 1 according to Embodiment 1. The imagingapparatus 11 further includes a communication unit 1103. Each composingelement of the imaging apparatus 11 is interconnected via bus. Acomposing element the same as the imaging apparatus 1 is denoted with asame reference sign, and description thereof is omitted.

The HMD2 according to Embodiment 3 includes an arithmetic unit 201, animage processing unit 202, a primary storage unit 204, a secondarystorage unit 205, a display unit 206, an operation unit 208 and athree-dimensional space recognizing unit 209, just like the imagingapparatus 1 according to Embodiment 1. The arithmetic unit 201 mayinclude a hardware circuit to implement the functions executed by theHMD2. The HMD2 further includes a communication unit 203. Each composingelement of the HMD2 is interconnected via a bus. A composing elementhaving a same name as the imaging apparatus 1 functions in a same manneras the corresponding composing element of the imaging apparatus 1, anddetailed description thereof is omitted. The display unit 206corresponds to the display unit 21 a and the display unit 21 b of theHMD2. The display unit 206 is an example of the “display control unit”.

The imaging apparatus 11 and the HMD2 communicate with each other viathe communication unit 1103 and the communication unit 203. The imagingapparatus 11 sends the image captured by the imaging unit 107 to theHMD2 via the communication unit 1103 and the communication unit 203.

FIG. 12A and FIG. 12B are flow charts exemplifying the image capturingsupport processing according to Embodiment 3. The image capturingsupport processing according to Embodiment 3 is a processing using theimaging apparatus 11 and the HMD2 in tandem. The image capturing supportprocessing according to Embodiment 3 includes image capturing processing1200 a in FIG. 12A performed by the imaging apparatus 11, and imagecapturing support processing 1200 b in FIG. 12B performed by the HMD2.

The image capturing processing 1200 a performed by the imaging apparatus11 is started when the user operates the operation unit 108 to set theimage capturing mode. The image capturing support processing 1200 bperformed by the HMD2 is started when the user operates the operationunit 208 to set the image capturing mode. The image capturing supportprocessing 1200 b may also be started when the imaging apparatus 11 isset to the image capturing mode, or when the captured image is receivedfrom the imaging apparatus 11.

The arithmetic unit 101 of the imaging apparatus 11 and the arithmeticunit 201 of the HMD2 execute programs for each composing element toperform each processing step of the image capturing processing 1200 aand the image capturing support processing 1200 b respectively.

In S1201 b, the three-dimensional space recognizing unit 209 of the HMD2acquires the position of the object and distance to the object in thereal space as the virtual space information, and holds the informationin the primary storage unit 204. The three-dimensional space recognizingunit 209 is an example of the “acquiring unit”. The virtual spaceinformation may be acquired in S1202 b after receiving the capturedimage from the imaging apparatus 1.

In S1202 a, the communication unit 103 of the imaging apparatus 1 sendsthe image captured by the imaging unit 107 to the HMD2 via thecommunication unit 203. In S1202 b, the communication unit 203 of theHMD2 determines whether the captured image was received from thecommunication unit 103 of the imaging apparatus 1.

If the captured image was received (S1202 b: Yes), processing advancesto S1203 b. If the captured image was not received (S1202 b: No), theprocessing step in S1203 b is repeated until the captured image isreceived.

In S1203 b, using the captured image received from the imaging apparatus1, the image processing unit 202 generates an imaging object, whichindicates an image capturing range where the imaging unit 107 iscapturing an image in the virtual space. The image processing unit 202adds the information on the generated imaging object to the virtualspace information that is held in the primary storage unit 204. Theimage processing unit 202 is an example of the “generating unit”.

In 51204 b, the image processing unit 202 generates a supporting objectin the same manner as S403 of the image capturing support processing 400according to Embodiment 1, or S902 of the image capturing supportprocessing 900 according to Embodiment 2. The image processing unit 202adds the information on the generated supporting object to the virtualspace information that is held in the primary storage unit 104.

In S1205 b, the display unit 206 of the HMD2 displays the imaging objectand the supporting object superimposed on the image capturing the realspace.

In S1206 b, the arithmetic unit 201 determines whether the userinstructed the operation unit 208 to end the image capturing mode. Ifthe end of the image capturing mode is instructed (S1206 b: Yes), theimage capturing support processing 1200 b ends. If the end of the imagecapturing mode is not instructed (S1206 b: No), processing returns toS1201 b, and the image capturing support processing 1200 b is repeated.

In S1206 a, just like S1206 b, the arithmetic unit 101 determineswhether the user instructed the operation unit 108 to end the imagecapturing mode. If the end of the image capturing mode is instructed(S1206 a: Yes), the image capturing processing 1200 a ends. If the endof the image capturing mode is not instructed (S1206 a: No), processingreturns to S1202 a, and the image capturing processing 1200 a isrepeated.

The image capturing support processing 1200 b may end when thenotification about the end of the image capturing processing 1200 a isreceived from the imaging apparatus 11, without determining the end inS1206 b. Further, the image capturing support processing 1200 b may endwhen a captured image is not received from the imaging apparatus 11 fora predetermined time.

In S1203 b of the image capturing support processing 1200 b, the imageprocessing unit 202 receives the captured image from the imagingapparatus 1, and acquires the position and orientation of the imagingobject from the received captured image, but the present invention isnot limited to this. The image processing unit 202 may acquire theposition and orientation of the imaging object by acquiring the positionand orientation of the imaging apparatus 11 from the image captured bythe imaging unit of the HMD2.

The imaging object and the supporting object are generated by the imageprocessing unit 202 of the HMD2, but may be generated via a cloud. Inthis case, the HMD2 communicates via the cloud and receives the imagingobject and the supporting object generated on the cloud, then displaysthe object superimposed on the display unit 206 of the HMD2. Whether theimaging apparatus 11 and the HMD2 are operated in tandem or not may beset in advance, or may be set when the user switches the mode of theimaging apparatus 11 to the imaging mode.

According to the imaging apparatus 11 and the HMD2 described above, theuser can support image capturing by utilizing the respective advantagesof the imaging apparatus 11 which specializes in imaging functions, andthe HMD2 which can display the AR objects in a wide range.

Other Embodiments

The imaging object is not limited to an AR object that indicates therange where an image is actually being captured. The image object mayalso be part of the image capturing range, instead of the entire imagecapturing range. In this case, the supporting object may be a part ofthe ideal image capturing range at the moving destination. The range ofthe supporting object may be determined such that ideal camera work canbe implemented by the user, overlaying the imaging object on thesupporting object. If the imaging object is a part of the imagecapturing range, the user can follow the imaging object to thesupporting object more accurately than the case of overlaying the entireimage capturing range on the supporting object.

A number of supporting objects is not limited to one, but may be two ormore. For example, the image processing unit 102 (image processing unit202) may generate a second supporting object based on the change amountof at least one of the position, orientation and moving speed betweenthe imaging object and the generated first supporting object. In thesame manner, the image processing unit 102 (image processing unit 202)may generate three or more supporting objects. If a plurality ofsupporting objects are displayed, the user can more easily estimate themoving destination of the imaging apparatus.

The predetermined position and the predetermined orientation of theimaging apparatus 1, to generate the supporting object, may be providedfrom an outside source. For example, the position and orientation of theimaging apparatus 1, when an image was captured in the past (e.g.several days or more in the past) using the imaging apparatus 1, may beprovided as the predetermined position and predetermined orientation togenerate the supporting object. Thereby the image capturing range of thecaptured image in the past can be superimposed and displayed as thesupporting object, and the user can capture an image at the sameposition and in the same orientation as the image captured in the past.

Each embodiment described above merely exemplifies a configuration ofthe present invention. The present invention is not limited to thespecific embodiments described above, and these embodiments may becombined or modified in various ways within the scope of the technicalspirit of the invention.

According to the present disclosure, an appropriate angle of view can beeasily determined, and image capturing in the three-dimensional spacecan be supported.

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2021-103950, filed on Jun. 23, 2021, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An electronic device for supporting imagecapturing by an imaging apparatus, the electronic device comprising atleast one processor and/or at least one circuit which function as: agenerating unit configured to generate a first object in a virtual spacecorresponding to a current image capturing range of the imagingapparatus, and a second object in a virtual space corresponding to animage capturing range in a case where the imaging apparatus has apredetermined position and a predetermined orientation; and a displaycontrol unit configured to display the first object and the secondobject.
 2. The electronic device according to claim 1, wherein thedisplay control unit displays the first object and the second object ina state of superimposing the first object and the second object on animage in which a real space is captured.
 3. The electronic deviceaccording to claim 1, wherein the generating unit determines thepredetermined position and the predetermined orientation, based on pathinformation which indicates a future movement of the imaging apparatus.4. The electronic device according to claim 3, wherein the pathinformation is inputted by a user.
 5. The electronic device according toclaim 3, wherein the path information includes at least any of: amovement to move the imaging apparatus straight at a constant speed; amovement to rotate the imaging apparatus in a horizontal directionaround the imaging apparatus or a user; a movement to rotate the imagingapparatus at a constant speed around an imaging axis of the imagingapparatus; and a movement to rotate the imaging apparatus around asubject.
 6. The electronic device according to claim 3, wherein based onthe path information, the generating unit generates a plurality of thesecond objects by regarding a position and an orientation of the imagingapparatus at each of a plurality of timings in the future as thepredetermined position and the predetermined orientation.
 7. Theelectronic device according to claim 1, wherein the generating unitdetermines the predetermined position and the predetermined orientation,based on changes between the position and orientation of the imagingapparatus in the past and the current position and orientation of theimaging apparatus, and generates the second object.
 8. The electronicdevice according to claim 7, wherein the generating unit furthergenerates a third object in a virtual space corresponding to an imagecapturing range in the case where the imaging apparatus has the positionand the orientation in the past, and the display control unit furtherdisplays the third object.
 9. The electronic device according to claim1, wherein the at least one processor and/or the at least one circuitfurther function as an acquiring unit configured to acquire informationon a virtual space by recognizing a position of an object and a distanceto the object in a real space.
 10. The electronic device according toclaim 1, wherein the generating unit generates the first object suchthat the position of the first object becomes any of: a position at afocal distance of the imaging apparatus; a position at a predetermineddistance from the imaging apparatus; and a position where a size of thefirst object becomes a predetermined size.
 11. The electronic deviceaccording to claim 1, wherein the generating unit generates the secondobject such that a relative relationship of the predetermined positionand the position of the second object becomes the same as a relativerelationship of the current position of the imaging apparatus and theposition of the first object.
 12. The electronic device according toclaim 1, wherein the display control unit displays the second object ina case where a difference of the positions or a difference of theorientations between the first object and the second object is athreshold or more.
 13. The electronic device according to claim 1,wherein the display control unit displays the second object in a casewhere a moving speed of the electronic device is not constant, or a casewhere a path of the movement of the electronic device is irregular. 14.The electronic device according to claim 1, wherein the imagingapparatus is integrated into the electronic device.
 15. The electronicdevice according to claim 1, wherein the imaging apparatus is separatedfrom the electronic device.
 16. The electronic device according to claim1, wherein the electronic device is a head mounted type display device.17. The electronic device according to claim 1, wherein the generatingunit requests an external device to perform processing to generate thefirst object and the second object, and acquires the first object andthe second object generated by the external device.
 18. A method forcontrolling an electronic device for supporting image capturing by theimaging apparatus, the method comprising: a generation step ofgenerating a first object in a virtual space corresponding to a currentimage capturing range of the imaging apparatus, and a second object in avirtual space corresponding to an image capturing range in a case wherethe imaging apparatus has a predetermined position and a predeterminedorientation; and a display control step of displaying the first objectand the second object.
 19. A non-transitory computer-readable mediumthat stores a program, wherein the program causes a computer to executea control method of an electronic device for supporting image capturingby the imaging apparatus, the control method comprising: a generationstep of generating a first object in a virtual space corresponding to acurrent image capturing range of the imaging apparatus, and a secondobject in a virtual space corresponding to an image capturing range in acase where the imaging apparatus has a predetermined position and apredetermined orientation; and a display control step of displaying thefirst object and the second object.